1. Field of the Invention
The present invention relates to equipment and methods for imaging a sample which may contain one or more species of luminous probes such as fluorescent probes, luminescent probes, quantum dot probes, up-converting probes, or other emissive probes; and more specifically, to equipment and methods which can image N species of probes with high optical efficiency, often requiring fewer than N observations of the sample.
2. Description of the Related Art
It is often of practical importance to image multiple luminous probes in a single sample, as in fluorescence in-situ hybridization (FISH), in chromatography plate readers, DNA sequencing, spectral karyotyping, general biological and neurobiological research, and the like.
One approach to multiprobe imaging, taken by Glass et. al. in U.S. Pat. No. 5,723,294 is simply to image the sample, in this case a multiwell plate, in several plate readers one after another. By suitable choice of probes and reader settings, unambiguous readings can be obtained from the combined results of all the readers. Yet this is hardly an integrated, efficient solution to the problem.
There is also a large literature of hyperspectral imaging, which can be used to obtain multiprobe images of samples. In hyperspectral imaging, several exposures of a sample are recorded using filters or interferometers in the optical path, from which an optical spectrum is derived for each point in the sample under study. Specific hardware for hyperspectral imaging includes filter wheels and circular-variable filters as in U.S. Pat. No. 5,591,981 and U.S. Pat. No. 5,784,152; angle-tuned interference filters as in the Renishaw imaging Raman microscope described in U.S. Pat. No. 5,442,438; acousto-optical tunable filters (AOTFs) as in U.S. Pat. No. 5,216,484, U.S. Pat. No. 5,377,003, and U.S. Pat. No. 5,556,790; optical interferometers as in U.S. Pat. No. 5,835,214, U.S. Pat. No. 5,817,462, U.S. Pat. No. 5,539,517, and U.S. Pat. No. 5,784,162; and liquid crystal tunable filters (LCTFs) as in Morris, et. al., xe2x80x9cImaging Spectrometers for Fluorescence and Raman Microscopy: Acousto-Optic and Liquid Crystal Tunable Filters,xe2x80x9d Applied Spectroscopy, 48:7:857-866, 1994. All of these except the interferometer systems are termed band-sequential systems, as each image records the entire spatial content of the sample, and successive images serve to step through its spectral content.
Alternatively, dispersive systems are used to obtain a spectrum for a single point or line, which is then scanned in two or one dimension respectively, to obtain a 2-D image of the sample with spectral data for each point. A non-dispersive system is described by Buican et. al., who use a photoelastic modulator (PEM) and polarizer in U.S. Pat. No. 4,905,169 to determine the spectral contents of a single point via the Fourier analysis of time-series intensity values at a detector; in U.S. Pat. No. 5,117,466 this arrangement is coupled with a laser scanning system to produce a two-dimensional image. Such systems are termed point-sequential or line-sequential imagers, as the entire spectral content is recorded more or less simultaneously, and successive readings step through the spatial content either pointwise or a line at a time.
Prior art describes fluorescence imaging where the excitation and/or emission selection is set to discrete wavelength settings (U.S. Pat. No. 5,784,152), and where the wavelength selection is continuously tunable (U.S. Pat. No. 5,591,981 and U.S. Pat. No. 5,863,504). Excitation light tuning is achieved by filter means such filter wheels, AOTF""s, LCTF""s; or via galvanometer-driven gratings; or via a series of paddle-mounted filters; or two arc lamps, each of which has control means for rapidly adjusting its intensity over a wide range (U.S. Pat. No. 5,491,343). In U.S. Pat. No. 5,208,651, Buican describes a method for time-encoding the excitation spectrum while concurrently analyzing the emission spectrum, via two PEM elements.
Normally, the individual bands used in hyperspectral imaging are distinct or nearly so, overlapping only in the transition region where a given band cuts off and the adjacent band cuts on. Each band has a transmission vs. wavelength response that approximates a steep-edged trapezoid, with sharp cut-on and cut-off, and approximately constant transmission through the passband. It is a universal goal in hyperspectral imaging to maximize the transmission at all wavelengths in the passband, as this yields increased signal-to-noise, which is a general concern in the field. Researchers have developed methods for broadening the inherently narrow bandpass of the AOTF, to obtain an approximately trapezoidal bandpass instead of a narrow sync function.
The prior art includes other methods for increasing throughput to obtain better signal-to-noise, such as the use of LCTF filters based on Solc designs, with broad bandpasses for increased throughput (Hoyt, xe2x80x9cTunable Liquid Crystal Filters Boost Fluorescence Imagingxe2x80x9d, BioPhotonics, July/August 1996). This may result in some overlap, or crosstalk, between adjacent spectral bands, which is dealt with by methods such as those described in the next three paragraphs. Notwithstanding the overlap between bands, this art is practiced with trapezoidal bandpass shapes or the like, having the highest practical transmission in each passband.
Many integrated multiprobe readers seek to take advantage of a priori knowledge of the samples being imaged. As the probes have more or less predetermined spectra, a complete spectrum may not be required. For example, it is not necessary to produce spectral data for those wavelength bands at which there is no possibility of optical emission. At the same time, the emission spectra of the various probes involved are not always distinct, but may also overlap to a considerable degree in some cases. If it is not possible to choose a set of wavelengths that correspond in a one-to-one fashion with the probes being imaged, the presence of emission at any given wavelength does not uniquely specify which probe was present. Rather, for a given experimental set-up, the observed energy ei at wavelength band xcexi is related to the concentration of the various probes cj according to:
ei=ai1*c1+ai2*c2+ai3*c3 . . . aiN*cNxe2x80x83xe2x80x83[1]
where coefficient aij specifies the optical radiation of probe j into wavelength band i.
This provides an easy way to determine the probe concentrations from the observed intensities, as follows. Equation [1] may be written in matrix form:
E=A*Cxe2x80x83xe2x80x83[2]
where E is the Mxc3x971 vector of observed energies at the M spectral bands, A is the Mxc3x97N matrix of terms aij, and C is the 1xc3x97N vector of probe concentrations. The matrix A has a direct physical interpretation. Each column corresponds to the spectrum of each particular probe, while each row corresponds to the emission of the various probes at a particular wavelength band. While some systems use a number of wavelength bands M greater than the number of probes N, it is common to use M=N, and to seek wavelengths where this matrix is approximately diagonal. Physically this means that for every probe, there is a corresponding wavelength band for which most, though not all, of the optical energy comes from that probe.
Since the number of wavelength bands M equals or exceeds the number of probes N, equation [2] may be inverted uniquely or in a least-squares error fashion to solve for the probe concentrations, viz.:
C=Axe2x88x921*Exe2x80x83xe2x80x83[3]
This allows direct calculation of the probe concentrations from the observed spectra, despite the presence of overlap or cross-talk between the spectral bands. Using this approach, Castleman used 3xc3x973 matrices to produce images of three fluorescent probes from three raw intensity images of sample emission, in xe2x80x9cColor Compensation for Digitized FISH images,xe2x80x9d Bioimaging, 1:159-165, 1993, and again in xe2x80x9cDigital Image Color Compensation with Unequal Integration Periods,xe2x80x9d Bioimaging, 2:160-162, 1994. The raw intensity images were obtained using a color CCD camera. In the BioPhotonics article cited above, Hoyt teaches the use of a fluorescence microscope with a monochrome CCD detector and an LCTF tuned to N bandpass settings in sequence, to image N probes, where N is typically from 3 to 5. Katzir et. al. also recite the benefits of this approach in U.S. Pat. No. 5,834,203.
However, all the foregoing imaging suffer from significant limitations. Imaging spectroscopic methods require complex and rather expensive hardware, and involve taking at least N exposures, using optical elements of relatively low efficiency. Multiple, long exposures are required, which limits throughput and is impractical with probes that photobleach rapidly. Alternatively, Castleman""s use of an RGB camera only resolves three probe species, with emission spectra that more or less correspond to the three primary colors.
In other prior art, photometric methods are known for determining the wavelength of a beam of light, using the differential spectral response of two detectors, or of two optical filters, from whose relative response to an optical signal the wavelength is determined. One such example is Van Der Gaag in U.S. Pat. No. 4,308,456, who uses two detectors coupled with optical filters that have more or less opposite spectral shapes, one with transmission that increases with wavelength and one with transmission that decreases. Garrett teaches in U.S. Pat. No. 5,627,648 how to determine wavelength of a beam of light by successive readings using photodiode with no filter, then with a plurality of filters in sequence, each having prescribed transmission vs. wavelength characteristics. Melle in U.S. Pat. No. 5,319,435, teaches the use of a Bragg-reflector element to create a controlled reflection vs. wavelength response, which is then combined with two photodetectors to provide a wavelength measurement system. Shih et. al. describe a system in U.S. Pat. No. 5,703,357 where light passes through a linear-variable filter onto two detectors having varying aperture ratio along the dispersion axis of the filter. The wavelength of light determines where along the linear variable filter the light passes through to the detectors, and thus determines the aperture ratio between the two detectors. From the ratio of relative response at the two detectors, the system determines the wavelength of incident light.
Gombocz et. al. describe a gel electrophoresis system in U.S. Pat. No. 5,410,412 and U.S. Pat. No. 5,104,512 which uses two photodetectors to view a gel electrophoresis plate, fed by separate fiber optics probes. Interposed in front of the two photodetectors are two filters, one that absorbs over the range 400-600 nm and one that absorbs over the range 500-700 nm. Using the relative response of the two filter-detector combinations and an unspecified Fourier analysis algorithm, the wavelength of light emitted by the gel plate is inferred; the purpose of Fourier methods is unclear. In U.S. Pat. No. 5,515,169, Cargill et. al. provide a wavelength-sensitive instrument similar to that of Gombocz except that the two optical fibers and two filters are replaced with a single beamsplitter that divides incident energy into two beams whose proportion varies linearly with wavelength. The resultant beams are sensed by two photodetectors, and the emission wavelength is determined from the ratio of detector signals via an unspecified algorithm.
Prober teaches in U.S. Pat. No. 5,306,618 a DNA sequencing instrument based on a two-channel photometer, where readings in the two channels indicate strength of sample emission in adjacent wavelength bands, each of which has a trapezoidal bandpass. This system is realized using two photodiodes or photomultiplier tubes, sensing the transmitted and reflected signals from a dichroic interference filter placed in the emission beam, which yields adjacent, complementary bands. Alternatively, Prober teaches dividing the sample emission into two beams, with a suitable bandpass filter and detector in each beam, to achieve the same result. The system is said to distinguish between four fluorescent probes having overlapping emission spectra. The degree of overlap is so great, and the photometer bands are so chosen, that at least three of the four probes, and possibly all four, have significant emission in each photometer band. From the sum of, and the ratio of, signal levels in the two bands, the presence of a probe and its species are determined. However, this system has several weaknesses. First, since the photometer bands are inherently much narrower than the emission spectrum of any given probe, only a small portion of the radiant energy from the sample is actually sensed. This leads to reduced optical sensitivity. Second, it is exquisitely sensitive to shifts in the wavelength of emission by the probes, as a shift of one-eighth of the emission width could lead to misidentification of which species was involved. Shifts of such magnitude can occur in multiprobe experiments, depending upon the chemical properties of the probe, the sample environment, quenching, pH, concentration, and the like. Yet there is no way to use this photometer with more widely-spaced probes, as it relies on the overlap of at least Nxe2x88x921 of the N species within each of the two photometer bands.
These photometric wavelength-detection systems are inherently non-imaging, in that they provide a wavelength indication for the whole beam of light, and cannot determine the wavelength at a plurality of separate points within an image. For gel chromatography or flow cytometry, this may be acceptable since the instrument insures that the material being sampled moves past the single point sensed by the photometer. However, in the vast majority of bio-medical applications, this is a severe limitation. Even in gel chromatography, this results in poor utilization of equipment. The plate reader is occupied for the entire duration of the gel separation, which can take hours; the alternative of a lengthy separation using an inexpensive instrument, followed by a rapid readout cycle in the more costly reader, is impossible.
Gombocz indicates that one can use mechanical means to scan the photometer to sense different regions across the surface of a gel plate. The same is true for the systems of Cargill, Var der Gaag, Buican, or Prober, in that these point-sensing photometric devices could be coupled with a 2-D scanner to provide an image of the sample. However, for this purpose there is little reason to choose these systems over alternatives such as e.g. a fiber-coupled diode-array spectrometer, which would provide more reliable spectral information with better utilization of the scarce emitted photons. The additional complexity of the scanner typically cancels out any innate simplification in the photometric wavemeter component. In short, when coupled to a scanner, the result is a low-performance hyperspectral imaging system, with no clear benefits over the prior art in that realm.
Nor do any prior-art wavelength-measuring systems provide means for directly producing a two-dimensional image with a two-dimensional sensor. Those which involve Fourier methods to determine wavelength are probably ill-suited to a taking a high-definition image, due to the computation requirements when the number of pixels is large. Buican, for example, describes in U.S. Pat. No. 4,905,169, the use of a phalanx of thirty-two computers, in parallel, to handle the data-processing requirements when his approach is used with single-element detectors at moderate data rates (150 xcexcs per sample reading).
These additional concerns relate to the special properties of imaging systems, which are not addressed by the prior art photometric systems. For example, all the photometric systems utilize ratios between two different measurements of a scene, via two detectors and a beamsplitter and perhaps additional filter elements; or via a single detector with various filters in time-sequence. Systems using two detectors face a significant cost burden over one-detector systems, as the cost of a two-dimensional imaging sensor such as a CCD is high. Either a one- or a two-detector system must provide means for spatially registering the images produced by the two detectors, or produced under various filter settings.
Indeed, this latter requirement is especially stringent since the heart of a photometric wavelength determination is a high-accuracy ratio of the two signal levels. Any error in the spatial registration of the multiple exposures degrades the numerical accuracy of the ratio scheme involved, since if readings from the two detectors come from different spatial regions in the sample, there is no way to take their ratio and determine a wavelength. In contrast, simple color imaging is relatively tolerant of minor mis-registration: the eye is very sensitive to fine spatial detail in the green band, but less so in the red and blue bands. Multi-detector color cameras need not achieve perfect registration between detector bands, yet can still produce an acceptable image. This is not true for an imaging system which yields a ratio-based photometric determination of wavelength, which must be registered to much better than one pixel.
Spatial registration of two detectors to a fraction of one pixel is a challenging and expensive proposition. Similarly, all known prior-art systems that use interference filters employ mechanical means to select between filters, or to engage them into and out of the beam. This mechanical switching leads to image shift from the unavoidable wedge in the filter elements, which corrupts the ratio used to derive a wavelength measurement.
Other practical considerations confound the extension of existing point-measuring photometric systems into imaging systems. The Prober photometer exhibits poor optical efficiency, since only a small portion of the sample emission is utilized by any given photometer band, even the most responsive band. This deficiency is even more problematic in an imaging system, where the desire for high spatial resolution favors the use of imaging detectors with many pixels; this in turn means that the radiant energy is partitioned between a great number of pixels, which exacerbates the signal-to-noise requirements.
In U.S. Pat. No. 4,833,332, Robertson Jr. teaches an improvement to the Prober photometer which is preferably constructed without any lenses or imaging optics, and uses non-imaging detectors such as photomultiplier tubes as detectors. Field-of-view restricting elements such as fiber-optic couplers maintain the desired passband of the filter elements. This arrangement is not capable of imaging a two-dimensional scene.
In another example, Buican""s system uses PEM elements to time-encode the spectral content of the beam, and the intensity of the encoded beam is read by a photodetector. Simply replacing the photodiode or photomultiplier tube (PMT) of Buican with a two-dimensional imaging detector such as a CCD, is impractical if not absolutely impossible. To do so would require continuously reading out the image from the CCD synchronously with the modulation by the PEM, which operate at frequencies in excess of 10,000 Hz (typically 50,000 Hz). On the other hand, CCD cameras capable of high-resolution imaging typically have sustained readout rates of 5 to 30 Hz, and even the most specialized high-speed digital cameras do not exceed 1,000 Hz.
In summary, the aforementioned art provides equipment and methods for hyperspectral imaging to produce high-definition images of multiple probes in a sample, but these require at least N observations to resolve N probes and use relatively expensive, optically inefficient hardware. Other systems provide wavelength measurements using photometric means, and require fewer observations or simultaneous observation with two detectors. However, these are point-measuring systems that cannot provide an image of the sample unless one adds mechanical or optical scanning means, which undermines the cost and performance benefits of this approach. And, attempts to extend this art to construct systems with two-dimensional imaging sensors are impractical for at least one of the following reasons in every case: the need for multiple imaging detectors; the need to register multiple images obtained from different detectors, or from a single detector with multiple filters, to much better than a single pixel; poor optical efficiency; the need for mechanical moving parts; intensive computing requirements; or, the need to operate at sample rates far above the capability of imaging detectors.
The first object of the present invention is to provide a system with high optical efficiency to achieve good signal-to-noise in detecting and discriminating between probes. A second object is to provide for imaging an entire two-dimensional scene at once, with an enormous advantage in throughput compared against the point-detectors or line-imaging systems of the prior art. It is a further object to provide means and methods for high-definition imaging of samples that may contain up to N species of luminous probes, in many cases using fewer than N observations. Yet another goal is to achieve these aims without any moving parts, to eliminate vibration and enhance reliability.
The invention uses filter elements, preferably liquid crystal type, to provide a few filter states, none of which have the conventional bandpass shapes as used in band-sequential multispectral imaging. One state may be an all-pass filter state, in which light of all wavelengths of interest is transmitted essentially without loss from the sample to the detector. A second state imposes a known spectral filter such as a downward- or upward-sloping response across the spectral band of interest. From the relative response of the detector to these two exposures, the wavelength of emission light, and hence the emitting probe species, is determined. While only two exposures are required, the system can distinguish between more than two species, with the resolution limit being set by the available probes and the signal-to-noise of the overall system.
A third exposure may be taken in a third filter state, which has a wavelength response that is linearly independent of the first two states, such as a triangle wave, a non-linear ramp, or a high-order sinusoid. Mathematically, this state is linearly independent from the first two states. Adding this exposure allows resolution of very densely-spaced probes, or enables unambiguous determination of probe species and proportions when a given pixel contains a mixture of two probes.
A key benefit is that in many instances fewer than N exposures are required in order to resolve N species. A second benefit is the high efficiency of the system. One of the filter states can be essentially a clear state, with transmission in excess of 80% if desired. The other filter states can have equally high peak transmissions, with transmission at other wavelengths depending on the filter state (ramp, triangle shape, and so on).
Another benefit is that the cost and complexity of the system are greatly reduced relative to prior-art LCTFs, AOTFs, or interferometers. While it is possible to construct the present invention using an interferometer as the filter element, in preferred embodiments the filter may consist of one or two filter stages as described in U.S. Pat. No. 5,892,612, xe2x80x9cTunable Optical Filter with White Statexe2x80x9d. Or, simple liquid crystal variable retarders between polarizers may employed instead. Generally, only one or two liquid crystal elements are required for the present invention, in contrast with prior-art LCTFs for multispectral imaging, which require from 5 to 12 elements. This reduces construction costs, and since the resulting assembly is typically only 3-6 mm thick, it is readily incorporated into microscopes, macroscopic imaging systems, and cameras without relay optics.
The use of liquid crystal elements as filters has several benefits. Unlike mechanically-scanned interferometric systems, there are no moving parts or precision optical surfaces, nor is the present invention sensitive to ordinary thermal drift. Unlike AOTF systems, no radio-frequency (RF) signals are required, and a large aperture is easily achieved. As with all LCTF imaging systems, diffraction-limited images may be readily obtained. This is a benefit in applications such as FISH, genetic therapy, functional imaging, chromosomal imaging, histological imaging, and the like.
Alternatively, one may use a filter wheel which is populated with filters made using colored glass, colored plastic, dielectric type filters, gels, and combinations of these. In a preferred embodiment of this type, one filter position is empty to provide a substantially clear state. Another filter provides a non-bandpass filter response such as a ramp or a periodic function of wavelength. It is easy to use software post-processing to spatially register images taken through each of the other filters relative to the clear state, as whatever features are present in a given filtered image must also be present in the unfiltered image, which transmits light from all probes. This eliminates the need for multiply-labeling certain species to produce a fiducial signal for registration purposes, as is required in the prior art. Even if no clear state is used, the fact that the filters have slope-type response functions, rather than bandpass-type functions, insures that several probes will be visible through any given filter, and vice versa. So again spatial registration is not a problem, and the prior-art limitation of needing a multiply-labeled fiducial is overcome.
Independent of the type of filter used, a further benefit is that the system obtains highly accurate data without requiring a high degree of uniformity in the imaging sensor employed. Also, the system can normally be used in a self-calibrating fashion, where the properties of the probes and filters are characterized in-situ and variations in chemistry are readily accommodated. Yet another benefit is that the system does not have burdensome computation requirements.
The present invention can be practiced in concert with a variety of imaging equipment, including confocal microscopes, epi- and darkfield illuminated microscopes, plate readers, high-throughput drug screening equipment, and other macroscopic imaging equipment. Acquisition at video rates is practical, which is a boon for high-throughput applications, or for the visualization of multiple probes in real-time.
It is possible to construct the present invention with cameras that have multiple-detectors, each of which views a distinct spectral band and makes species determinations for species emitting in that band, based on ratios of intensity in the various filter settings. This allows imaging plural colocalized probes in a very light-efficient manner. No extraordinary registration of one imaging detector to another is required, since the ratio used for the photometric determination of species is the ratio of two readings from a given detector, rather than one detector compared to another. This embodiment can be practiced with conventional three-CCD RGB cameras as well as custom designs to access other spectral bands.
Other objects and features of the present invention will become apparent from the following detailed description considered in conjunction with the accompanying drawings. It is to be understood, however, that the drawings are designed solely for purposes of illustration and not as a definition of the limits of the invention, for which reference should be made to the appended claims.